Comparison of Gaussian process modeling software

Gaussian process fitting, or kriging, is often used to create a model from a set of data. Many available software packages do this, but we show that very different results can be obtained from different packages even when using the same data and model. We describe the parameterization, features, and optimization used by eight different fitting packages that run on four different platforms. We then compare these eight packages using various data functions and data sets, revealing that there are stark differences between the packages. In addition to comparing the prediction accuracy, the predictive variance--which is important for evaluating precision of predictions and is often used in stopping criteria--is also evaluated

Data from fitting Gaussian process models to various data sets using eight Gaussian process software packages

The article of record as published may be found at http://dx.doi.org/10.1016/j.dib.2017.12.012This data article provides the summary data from tests comparing various Gaussian process software packages. Each spreadsheet represents a single function or type of function using a particular input sample size. In each spreadsheet, a row gives the results for a particular replication using a single package. Within each spreadsheet there are the results from eight Gaussian process model-fitting packages on five replicates of the surface. There is also one spreadsheet comparing the results from two packages performing stochastic kriging. These data enable comparisons between the packages to determine which package will give users the best results.Office of Naval Research via NPS's CRUSERNaval Supply Systems Command Fleet LogisticsGrant number N00244-15-2-000

Sliced Full Factorial-Based Latin Hypercube Designs as a Framework for a Batch Sequential Design Algorithm

The article of record as published may be found at http://dx.doi.org/10.1080/00401706.2015.1108233When fitting complex models, such as finite element or discrete event simulations, the experiment design should exhibit desirable properties of both projectivity and orthogonality. To reduce experimental effort, sequential design strategies allow experimenters to collect data only until some measure of prediction precision is reached. In this article, we present a batch sequential experiment design method that uses sliced full factorial-based Latin hypercube designs (sFFLHDs), which are an extension to the concept of sliced orthogonal array-based Latin hypercube designs (OALHDs). At all stages of the sequential design, good univariate stratification is achieved. The structure of the FFLHDs also tends to produce uniformity in higher dimensions, especially at certain stages of the design. We show that our batch sequential design approach has good sampling and fitting qualities through both empirical studies and theoretical arguments. Supplementary materials are available online.USMC-PMMIONR/NPS CRUSE

Gradient Based Criteria for Sequential Design

Computer simulation experiments are commonly used as an inexpensive alternative to real-world experiments to form a metamodel that approximates the input-output relationship of the real-world experiment. While a user may want to understand the entire response surface, they may also want to focus on interesting regions of the design space, such as where the gradient is large. In this paper we present an algorithm that adaptively runs a simulation experiment that focuses on finding areas of the response surface with a large gradient while also gathering an understanding of the entire surface. We consider the scenario where small batches of points can be run simultaneously, such as with multi-core processors

Improving the efficiency and efficacy of controlled sequential bifurcation for simulation factor screening

Controlled sequential bifurcation (CSB) is a factor-screening method for discrete-event simulations. It combines a multistage hypothesis testing procedure with the original sequential bifurcation procedure to control both the power for detecting important effects at each bifurcation step and the Type I error for each unimportant factor under heterogeneous variance conditions when a main-effects model applies. This paper improves the CSB procedure in two aspects. First, a new fully sequential hypothesis-testing procedure is introduced that greatly improves the efficiency of CSB. Moreover, this paper proposes CSB-X, a more general CSB procedure that has the same error control for screening main effects that CSB does, even when two-factor interactions are present. The performance of the new method is proven and compared with the original CSB procedure

The effects of common random numbers on stochastic kriging metamodels

Ankenman et al. introduced stochastic kriging as a metamodeling tool for representing stochastic simulation response surfaces, and employed a very simple example to suggest that the use of Common Random Numbers (CRN) degrades the capability of stochastic kriging to predict the true response surface. In this article we undertake an in-depth analysis of the interaction between CRN and stochastic kriging by analyzing a richer collection of models; in particular, we consider stochastic kriging models with a linear trend term. We also perform an empirical study of the effect of CRN on stochastic kriging. We also consider the effect of CRN on metamodel parameter estimation and response-surface gradient estimation, as well as response-surface prediction. In brief, we confirm that CRN is detrimental to prediction, but show that it leads to better estimation of slope parameters and superior gradient estimation compared to independent simulation